Skip to content

Definition-first projects, the Map workspace, data tools, training and curation - #126

Merged
MrScripty merged 149 commits into
mainfrom
feat/definition-first-projects
Nov 19, 2026
Merged

MrScripty merged 149 commits into
mainfrom
feat/definition-first-projects

Conversation

@MrScripty

Copy link
Copy Markdown
Owner

Brings Tuldok's November work to main (148 commits).

  • Definition-first projects: a project is a Git folder of definitions and high-level results; PostgreSQL is the working copy; weights and products live in a per-project directory on the machine. The old project mode is removed.
  • The Map workspace (Map, Data, Annotate, Train, Sweeps, Benchmarks, Compare, Analysis, Overview), updating in place.
  • Data tools: looking inside any product (text, image, audio, point cloud, mesh, arrays), weights by layer, annotation with tasks in Git, Import/Freeze/Generate, the Pumas 0.8.0 client and endpoint, kept fields (native, predicted, signed error).
  • Training and curation: live training views and tuldok_log (autolog, export_graph), the model structure viewer, Compare and parallel coordinates with bootstrap intervals, generated annotation with agreement and review, curation by embeddings (EmbeddingGemma 2 default), serving a model on this machine, and hyperparameter sweeps.
  • Plans, specs and the execution ledger are under docs/plans/; docs/plans/training-and-curation/comparison.md compares this with the state before the last plan, and docs/research/competitive-landscape/ compares Tuldok with other tools.

Checks on the final code: project suite 898 tests and agent interface 43 (4 skipped) pass; all 15 workspace browser checks pass.

Includes the work of #125 (Pumas 0.8 typed contract).

🤖 Generated with Claude Code

MrScripty and others added 30 commits November 2, 2026 09:59
Pumas v0.8 (feat/v080-consumer-api, 4c79cd8e) grows typed responses
additively and makes error.outcome authoritative. Tuldok refused valid
responses and misreported refusals.

- Capabilities: validate only the fields, capabilities (chat, image
  generation, image-to-text) and option bounds Tuldok uses; ignore
  context_tokens, multimodal_embedding/decision, new formats, reasons,
  options and fields; select by name; drop list-length caps (1 MiB body
  bound instead).
- Errors: classify by error.outcome, not HTTP status (404/409 are
  ordinary refusals); accept request_id null; keep unknown codes; show
  Pumas' bounded error.message in job/UI errors, capability refusals and
  pumas_error bodies on OpenAI-compatible routes (after redaction).
- Results: ignore additive envelope/result fields.
- Image-to-text: no fixed 2048 max_tokens cap; the live bound follows
  the served context. Image facts are checked only when advertised.
- Model listing prefers GET /v1/served-models (profiles shown), falling
  back to /v1/models on 404.
- Repin the six actual DTO schema exports, source-derived capability
  fixture and source.json to 4c79cd8e; update the consumer contract doc.
  The PR51 advertisement decoder stays closed because it is retained as
  evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ement

The gateway advertisement decoder was closed: any field Pumas added to
the advertisement, or more than 64 entries in an open list, made the
gateway disappear from scans. Required fields are now checked for
presence and type only. Unknown fields are ignored and dropped from the
decoded advertisement, so the stored build observation keeps only the
known fields; list-length caps give way to the 64 KiB body bound.

Missing required fields, wrong types, unsupported schema versions and
malformed endpoints still fail, now as descriptor.Rejected with a
reason naming the field. A gateway scan returns refused advertisements
under `rejected` with their reasons, and the chooser shows them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The vision gateway fixture is projected from the Pumas DTOs, which now
also advertise decision and multimodal_embedding. Pumas responses grow
additively, so the browser flow asserts image_to_text is offered rather
than an exact capability count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The approved redesign: a project folder holds only definitions, notes and
result summaries; what is built from them stays in the database and a local
product store. Includes the implementation spec for milestones 1 and 2 and
the execution ledger for milestone 1.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Adds the tuldok_project package: strict reading of a project folder, JSON
Schema and cross-file checks for the seven definition kinds, the uses-graph,
and benchmark result summaries (statistics and results/*.csv files).
Command line: python -m tuldok_project check FOLDER.

No existing module is changed or imported except jsonschema.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A reference for people and agents who write a definition-first project
folder: the layout, the seven definition kinds, result summary files, the
rules that span files, and how to check a folder. Every example in it
passes python -m tuldok_project check.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
store.open_project imports a checked folder into one schema per project
(definitions as exact text plus jsonb, and the uses-graph as an edges
table), attaches without re-checking when no definition file changed, and
refuses a folder with errors without touching the database. Definitions can
be set, removed and saved back to their files, with conflicts between
unsaved edits and files changed on disk refused unless told to keep or
discard. graph.states derives each definition's state from its own label
and what it uses.

The lab's 263 definitions open in 0.6 s into an empty database and in
0.01 s when nothing changed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
python -m tuldok_project gains open, list, show, graph, status, set, remove,
save and forget on top of the store. Each command opens the project (which
imports any change on disk), does its work and releases the project's lock
before it exits; reading commands take --json. The helpers the store shares
with the reader and the checker are public names now.

The contract describes the commands, and the ledger records milestone 2:
the lab's 263 definitions open in under a second.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… milestone 3

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Results files are imported on open into a summary table, and are part of
what makes a folder changed. A run's per-case values (tuldok_results_v1)
can be imported; summaries for that subject are then computed from them,
one overall row and one row per single group value, and save writes
results/<benchmark>.csv back when it differs. Reading summaries, values and
the overview is SQL only.

On the lab project, 478,660 values import in 12 s and the recomputed
summaries equal the files the lab's build script wrote, byte for byte.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
charts.chart_data answers a chart definition (ranking, tradeoff,
progression, comparison) from the summary table with SQL only; a test
makes every file read fail while it runs. render_svg draws the data as a
self-contained SVG in the workspace design tokens, light or dark, and
render_report builds one HTML page for a project: where it stands, every
chart, the results overview and the definitions by kind.

Command line: chart and report; results, values and chart accept a bare
name or a reference. The contract documents results, values files, the
chart forms and the commands.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
plan.build_plan lists, in dependency order, the steps that build one or
more definitions: environments to set up, steps to build with the GPU,
disk and time their definitions declare, and blocked targets with the
definitions that cause the block and their reasons. Nothing is followed
out of a blocked step, and a benchmark names subjects it would skip.
Making a plan reads the database only; a test makes process creation and
file opening fail while it runs.

Command line: python -m tuldok_project plan FOLDER TARGET... [--all].

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
An environment definition can be fulfilled on this machine by connecting
an existing interpreter or by installing one with uv from the definition's
requirements. The check runs the interpreter once, reads the Python and
package versions without importing them (torch is imported for a cuda
environment to confirm the GPU is usable), and stores the result, so an
environment is not set up, ready or failing. Plans show a ready
environment as ready, not as work to do.

Command line: environments, connect, check-environment, set-up (with
--dry-run and --replace) and disconnect.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
snapshot.prepare copies the code a run needs from HEAD into a read-only
directory under the project's local home, once per commit and set of code
paths, and refuses a tree with uncommitted changes to tracked files unless
a scratch run is asked for. tuldok.json gains code_paths, the parts of the
repository a run needs. changed_since lists what changed after a commit,
for telling when a product is out of date.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…gone

launcher.run starts a command in its own session with its run id in the
environment, watches wall time, disk use and a cancel file, stops it with
SIGTERM then SIGKILL, and afterwards finds and kills anything still
carrying the run id, reaping what it adopts as a subreaper. strays and
stop_strays do the search alone for a run whose launcher is gone, and
progress reads the log tails and the last tuldok_progress_v1 line.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
launcher.run now starts a small custodian (python -m tuldok_project.launcher)
that is the child subreaper for one run: it starts the command, watches and
stops it, and reaps everything it adopts before writing the outcome. The
calling process no longer becomes a subreaper, so it is left with no
zombie children and its own unrelated children keep their exit status. A
run is stopped and swept when its caller dies, and a custodian that
disappears without an outcome gives unknown after a sweep by run id.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
builds.execute runs a plan's steps in order from the commit's snapshot
through the launcher: it resolves placeholders, uses the environment's
checked interpreter, records each run and its product, checks declared
outputs, and imports a benchmark's results. The project lock is released
while a command runs, and a GPU step holds a machine-wide lock.

A definition is now running, built, out of date or failed: out of date
when its definition, its settings file, an input's product or its code
changed since the run, or when something it uses is out of date. Plans
show built steps as built and say why a step is out of date.

Command line: build (with --scratch, --yes and --env), runs, products.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…imates

Adds run (one run's details), progress (the last progress line and the log
tails, with --follow), cancel (asks a running run to stop), and stop
(kills what a run left behind when its builder is gone). A run can be
named by an unambiguous prefix of its id.

estimate runs each step's estimate command from the commit's snapshot
under the launcher and stores the figures; plan shows them beside the
declared ones while the definition is unchanged. It is the only way an
estimate command is ever run.

The contract gains a Building section covering snapshots, runs, products,
states, limits, the commands and estimates.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
examples/tiny-project is the smallest real project for the new format: a
dataset of noisy sine curves, a three-layer network trained on a GPU in
under a minute, a benchmark against a predict-zero baseline, and two
charts. Its README gives the commands to connect two environments, plan,
build, follow progress and draw the charts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
One project folder per Tuldok process, every operation a POST with its
contract selected, builds asked for through the interface as requests the
owner approves, and what was built differently recorded in section 12.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…quests

operations.py declares the 36 operations a served folder offers: who may
call each, its arguments and its result keys. service.py checks a call
against that table, opens the project only for the moment it needs, and
records every change asked for in a new activity table.

A build asked for through the service is a build request. The owner's is
queued at once; an agent's waits for the owner's approval, which holds to
the commit and to the text of every definition in the plan. One worker per
project runs queued requests in turn; builds.execute gains expect and stop
so a changed definition or commit stops the build before the step starts.

The command line gains requests, approve and decline, and its commands now
wait a few seconds for a project that the app or a build has open.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
python app.py --project-folder PATH serves one definition-first project:
POST /api/project/<name> for each operation, the project_folder capability
profile, an owner client and the MCP bridge flag for it, and a page that
shows the project's report.

The catalog and old project capability manifests and the legacy discovery
document are byte for byte unchanged; a test pins their digests. The agent
client now passes on a project refusal's code and problems.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
MrScripty and others added 25 commits November 19, 2026 07:00
…d records

Per type: kappas, Kendall's tau, interval IoU, matched IoU (boxes, polygons
on a grid, rotated 3-D boxes, point sets, instance masks; Hungarian up to 64
objects a term, greedy above), semantic mask IoU, OKS, depth difference,
track IoU with identity switches and fragments; a confidence filter and the
disputed list. Tested against the test's own arithmetic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, and the decision kept

annotation_sets, annotation_agreement, annotation_compare and keep_annotation
(spec-t4 section 6), in the project_folder profile. A decision goes through
set_annotation's own path; a kept sidecar is staged in the working copy and
written to annotations/<task>/ by Save, which also removes those no saved
line names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nnotation type

Two of a task's sets compared side by side or overlaid: masks, polygons,
boxes, keypoints and depth drawn on the image, point objects and 3-D box
wireframes on the point cloud, stretches under a series or a sound, tracks
on an array's frames with a slider. The agreement, the most disputed
records, Keep A / Keep B / Keep both / Reject both through keep_annotation,
and a light correction of kept polygons and point objects. Parts are put
in place only when they differ. New task offers the nine new types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd the browser check of review

examples/tiny-project gains make_scenes.py (about 4 KB of images, point
clouds, frames and a series), a stand-in annotator, two tasks and four
annotators. browser_project_workspace_annotation_kinds.cjs builds a task of
eight types with two stand-ins each and checks the overlays, the agreement
and disputed order, a decision kept and saved, both corrections, updates in
place, light and dark at 1440 and 390.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t embedder

Also: the tab bar no longer widens the page (the Compare tab made it 65 px too
wide at 1000 px); on a phone the tabs take a scrolling row of their own; the
annotate check expects the Compare tab.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…limits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… served models

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… behind Tuldok's credentials

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tab and its drawer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, agreement and review

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s credentials

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l after another

A sweep definition (sweeps/<name>.json) names a recipe, the settings it
varies, a strategy (grid, random, successive halving), one or two
objectives from the trials' training logs and a budget. A build request
for it alone covers the sweep, the recipe and what the recipe needs; the
worker runs each trial as a derived recipe kept in the working copy,
stops one that halving prunes or that outlasts the minutes, and records
the runs and trials. Two reads, sweeps and sweep, and the map's groups
of trials.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…parallel coordinates and on its frontier

New sweep writes a sweep's text from a recipe's settings; the state block
(run, trials by how they ended, budget used, running trial, best so far)
is set in place on every reading; Approve and Decline when a request
awaits approval; the trials table opens a trial in the drawer or on the
Map, where trials carry the kind of change Sweep trial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Also: the annotate check expects the Sweeps tab.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the lab's 206 recipes it asked definition() per recipe, which works out the
whole project's states each time: 13 s. Now 0.09 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… full width

In the 420 px drawer only two or three operations fit across. Open wide draws
the same block, with the same state, over the page; Escape or Back to the
drawer closes it. The structure check opens and closes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 52f65f08-a579-4077-be1c-771322a7ea66
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

On CI the environment has NumPy, so predict receives a float32 array and the
answer encodes it again; the test assumed lists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MrScripty
MrScripty merged commit d8a57a1 into main Nov 19, 2026
5 checks passed
@MrScripty
MrScripty deleted the feat/definition-first-projects branch November 19, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant